CMOS imaging sensors are commonplace in many handheld devices. One of the challenges of this technology is the image flicker that is caused by the interaction of fluorescent lighting and the sensor's electronic rolling shutter. Non stationary sequences pose a challenging problem to multi frame processes. To address these limitations a new process is proposed to improve flicker detection performance by using frame alignment. This method significantly improves the performance in handshake and vertical panning conditions.
Imaging devices are increasingly commonplace in many digital handheld devices. Many of these devices use CMOS imaging sensors with an electronic rolling shutter. The rolling shutter unlike a mechanical shutter requires that different parts of the image are exposed over different time intervals. Specifically, each successive row of the image is offset in time. One consequence of this approach is that spatial flicker is generated in the image under fluorescent lighting since each row is exposed to a different amount of total illuminance. Robust detection of this flicker is essential in order to properly configure the camera to remove the artifacts. A real time imaging system places additional constraints on flicker detection. Practical imaging systems need to operate at 15 or 30 fps which greatly reduces the spatial flicker frequencies in the image making the detection process more challenging.
Various approaches to this problem have been proposed. Baer et. al. proposed modifying the sensor architecture to read out each pixels twice in opposite orders and sum the result to reduce the appearance of flicker as described in R. L. Baer, R. Kakarala, “Systems and methods for reducing artifacts caused by illuminant flicker,” U.S. Pat. No. 7,397,503. Other methods use the captured image or image sequence to detect flicker. Many such methods rely on sequential frames in order to detect flicker or described in D. Poplin, “An Automatic Flicker Detection Method for Embedded Camera Systems,” IEEE Transactions on Consumer Electronics, 52(2), 308-311 (2006); T. Tajbakhsh, R. R. Grigat, “Illumination Flicker Frequency Classification in Rolling Shutter Camera Systems,” Proceedings of IASTED 2007, Signal and Image Processing, 288-293 (2007); and M. Kaplinsky, I. Subbotin, “Method for mismatch detection between the frequency of illumination source and the duration of optical integration time for imager with rolling shutter,” U.S. Pat. No. 7,142,234. In these works the scene is always assumed to be stationary which in practice is rarely the case.
It has been found that the illuminance from a fluorescent light source varies in time due to the alternating current supply in the power source. Depending on the geographical location the supply is either 60 or 50 Hz. The power transferred to the light source oscillates at twice the supply frequency resulting in an illuminance that varies at either 120 or 100 Hz. The illuminance from the light source is modeled by the following function:
                              s          ⁡                      (            t            )                          =                              A            +                          B              ⁢                                                          ⁢                                                sin                  2                                ⁡                                  (                                                                                    2                        ⁢                        π                                                                    2                        ·                        T                                                              ⁢                    t                                    )                                                              =                      A            +                          B              2                        -                                          B                ⁢                                                                  ⁢                                  cos                  ⁡                                      (                                                                                            2                          ⁢                          π                                                T                                            ⁢                      t                                        )                                                              2                                                          (        1        )            
Where A is the DC illuminance at the zero crossing of the voltage in the power source when no power is being transferred to the light source. The DC portion of the illuminance does not contain any flicker so for simplicity A is set to zero. B is a scalar that determines the magnitude of flicker in a given light source and is equal to the illuminance when the voltage is at a maximum minus the constant A. Both A and B are constants that will vary from light source to light source. T is the flicker period. Flicker at this frequency is not perceived by human vision however a rolling shutter CMOS sensor can pick up this variation.
Consider FIG. 1 which plots Equation 1 for F=120 Hz and A=0, B=1 for simplicity. A typical electronic rolling shutter is shown below it. In a rolling shutter, each row starts and finishes integrating at different moments in time. The difference between any two rows is given by Δr. The signal in row 1 is proportional to the area under the curve over its integration time shown with the descending hatching. Accordingly, the signal in row n is proportional to the area shown with rising hatching. The cross hatched area corresponds to the time over which both row 1 and row n are integrating together. This figure illustrates that the total area for row 1, and thus the signal for row 1, is greater than the area for row n. This relative difference in exposure manifests itself as dark to light banding over the image. Not all integration times will produce flicker. If the integration time is set in multiples of the flicker frequency then the area under the curve is constant irrespective of the offset Δr. In such cases no flicker is present. Integration time is therefore expressed by the following equation:ti=N·T+Δt  (2)
Where N is the number of full flicker periods and Δt is difference in time between N flicker periods and the actual integration time.
The sensor response of the each row in the sensor is proportional to the illuminance that each row receives over the integration time. It is possible to derive the sensor response to the flicker illuminance as a function of the offset in row start time Δr, and integration time ti.
                                                        R              n                        ∝                                          ∫                                  t                  sn                                                  t                  en                                            ⁢                                                s                  ⁡                                      (                    t                    )                                                  ⁢                                                                  ⁢                                  ⅆ                  t                                                              =                                                    ∫                                  Δ                  ⁢                                                                          ⁢                  r                                                                      N                    ·                    T                                    +                                      Δ                    ⁢                                                                                  ⁢                    t                                    +                                      Δ                    ⁢                                                                                  ⁢                    r                                                              ⁢                                                s                  ⁡                                      (                    t                    )                                                  ⁢                                                                  ⁢                                  ⅆ                  t                                                      =                                                            ∫                                      Δ                    ⁢                                                                                  ⁢                    r                                    T                                ⁢                                                      s                    ⁡                                          (                      t                      )                                                        ⁢                                                                          ⁢                                      ⅆ                    t                                                              +                                                ∫                  T                                      N                    ·                    T                                                  ⁢                                                      s                    ⁡                                          (                      t                      )                                                        ⁢                                                                          ⁢                                      ⅆ                    t                                                              +                                                ∫                                      N                    ·                    T                                                                              N                      ·                      T                                        +                                          Δ                      ⁢                                                                                          ⁢                      r                                                                      ⁢                                                      s                    ⁡                                          (                      t                      )                                                        ⁢                                                                          ⁢                                      ⅆ                    t                                                              +                                                ∫                                                            N                      ·                      T                                        +                                          Δ                      ⁢                                                                                          ⁢                      r                                                                                                  N                      ·                      T                                        +                                          Δ                      ⁢                                                                                          ⁢                      r                                        +                                          Δ                      ⁢                                                                                          ⁢                      t                                                                      ⁢                                                      s                    ⁡                                          (                      t                      )                                                        ⁢                                                                          ⁢                                      ⅆ                    t                                                                                      ⁢                                                      (        3        )                                          =                                    N              ⁢                                                ∫                  0                  T                                ⁢                                                      s                    ⁡                                          (                      t                      )                                                        ⁢                                                                          ⁢                                      ⅆ                    t                                                                        +                                          ∫                                  Δ                  ⁢                                                                          ⁢                  r                                                                      Δ                    ⁢                                                                                  ⁢                    r                                    +                                      Δ                    ⁢                                                                                  ⁢                    t                                                              ⁢                                                s                  ⁡                                      (                    t                    )                                                  ⁢                                                                  ⁢                                  ⅆ                  t                                                                    ⁢                                                      (        4        )            
Substituting Eqn. 1 with A=0 into Eqn. 4 gives the following:
                    =                                            B              2                        ⁡                          [                                                N                  ⁢                                                            ∫                      0                      T                                        ⁢                    1                                                  -                                                      cos                    ⁡                                          (                                                                                                    2                            ⁢                            π                                                    T                                                ⁢                        t                                            )                                                        ⁢                                                                          ⁢                                      ⅆ                    t                                                  +                                                      ∫                                          Δ                      ⁢                                                                                          ⁢                      r                                                                                      Δ                        ⁢                                                                                                  ⁢                        r                                            +                                              Δ                        ⁢                                                                                                  ⁢                        t                                                                              ⁢                                                                          ⁢                  1                                -                                                      cos                    ⁡                                          (                                                                                                    2                            ⁢                            π                                                    T                                                ⁢                        t                                            )                                                        ⁢                                      ⅆ                    t                                                              ]                                ⁢                                          =                                    B              2                        ⁡                          [                              NT                +                                  Δ                  ⁢                                                                          ⁢                  t                                +                                                      T                                          2                      ⁢                      π                                                        ⁡                                      [                                                                  sin                        ⁡                                                  (                                                                                                                    2                                ⁢                                π                                                            T                                                        ⁢                            Δ                            ⁢                                                                                                                  ⁢                            r                                                    )                                                                    ⁢                                                                                          -                                              sin                        ⁡                                                  (                                                                                                                                                      2                                  ⁢                                  π                                                                T                                                            ⁢                              Δ                              ⁢                                                                                                                          ⁢                              r                                                        +                                                                                                                            2                                  ⁢                                  π                                                                T                                                            ⁢                              Δ                              ⁢                                                                                                                          ⁢                              t                                                                                )                                                                                      ]                                                              ]                                                          (        5        )            
This equation describes the response of a row offset by Δr. This equation can be generalized to describe the response of row n by using the following relation.Δr=nΔrd  (6)
Where Δrd is the row delay between two adjacent rows. The following equation of row response as a function of row number results from substituting Eqn. 6 into Eqn. 5.
                              R          ⁡                      (            n            )                          =                              B            ⁡                          [                              NT                +                                  Δ                  ⁢                                                                          ⁢                  t                                +                                                      T                                          2                      ⁢                      π                                                        ⁡                                      [                                                                  sin                        ⁡                                                  (                                                                                                                    2                                ⁢                                π                                                            T                                                        ⁢                            n                            ⁢                                                                                                                  ⁢                            Δ                            ⁢                                                                                                                  ⁢                            rd                                                    )                                                                    ⁢                                                                                          -                                              sin                        ⁢                                                  (                                                                                                                                                      2                                  ⁢                                  π                                                                T                                                            ⁢                              n                              ⁢                                                                                                                          ⁢                              Δ                              ⁢                                                                                                                          ⁢                              rd                                                        +                                                                                                                            2                                  ⁢                                  π                                                                T                                                            ⁢                              Δ                              ⁢                                                                                                                          ⁢                              t                                                                                )                                                                                      ]                                                              ]                                =                      B            ⁡                          [                              NT                +                                  Δ                  ⁢                                                                          ⁢                  t                                -                                                      T                    π                                    ⁡                                      [                                                                  cos                        ⁡                                                  (                                                                                                                    2                                ⁢                                π                                                            T                                                        ⁢                            n                            ⁢                                                                                                                  ⁢                            Δ                            ⁢                                                                                                                  ⁢                            rd                                                    )                                                                    ⁢                                                                                          ·                                              sin                        ⁡                                                  (                                                                                    π                              T                                                        ⁢                            Δ                            ⁢                                                                                                                  ⁢                            t                                                    )                                                                                      ]                                                              ]                                                          (        7        )            
In the final model, the response contains a linear term B(NT+Δt) which is determined solely by the integration time and is independent of the row. From that a modulated harmonic term is subtracted. In this term a cosine harmonic is a function of row number and the row delay. This harmonic is modulated by a sinusoid whose frequency depends on Δt. The effect of this modulation is clear. When the integration time is a multiple of the flicker frequency Δt=0 and Eqn. 7 reduces to the following.R=BNT  (8)
As expected the response no longer varies between rows and is only dependant on the length of integration time. In such a case, no flicker will be present in the image. It is also important to note that the modulation factor will be maximum when Δt=T/2. As will be shown, the magnitude of the flicker signal is generally weak so maximizing the amplitude of the signal will aid detection.
The row delay Δrd will affect the frequency of the flicker in the image. FIG. 2 plots the theoretical response at three different row delays. As the delay between adjacent rows increases, the frequency of the flicker seen in the image also increases. The relationship between the flicker period of the light source and the spatial period of the flicker in the image is:
                              T          spatial                =                              T            flicker                                Δ            ⁢                                                  ⁢            rd                                              (        9        )            
One of the significant challenges of a real time flicker detection system is the low spatial frequency of the flicker in an image. At 30 fps the frame time is only 33.33 μs which means there are only 3.3 periods at 50 Hz and 4 periods at 60 Hz.
The total integration time also affects the signal strength of flicker in the image. The flicker ratio Fratio is defined as the response from the row with the maximum signal divided by the response from the row with the minimum signal.Fratio=Max [R(n)]/Min [R(n)]  (10)
FIG. 3 plots this ratio as a function of total integration time for a 60 Hz flicker frequency. The ratio moves towards unity at multiples of the flicker frequency and with peaks when Δt=T/2. However, the peaks get smaller as N increases. The flicker ratio and therefore the flicker in the image increases towards infinity for integration times less than T.
Previous works have demonstrated that scene content in the image can be reduced significantly by subtracting successive frames such as D. Poplin, “An Automatic Flicker Detection Method for Embedded Camera Systems,” IEEE Transactions on Consumer Electronics, 52(2), 308-311 (2006); and M. Kaplinsky, I. Subbotin, “Method for mismatch detection between the frequency of illumination source and the duration of optical integration time for imager with rolling shutter,” U.S. Pat. No. 7,142,234. The results were obtained using static scenes and did not consider more practical cases were the scene content is dynamic. Since the magnitude of the flicker signal can be quite small relative to the signal of the underlying scene even a small misalignment will cause signals from the scene to remain after subtraction. Previous works have dealt with the problem of frame to frame alignment to reduce shearing effects of the rolling shutter as set forth in C. K. Liang, L. W. Chang, H. H. Chen, “Analysis and Compensation of Rolling Shutter Effect,” IEEE Transactions on Image Processing, 17(8), 1323-1330 (2008). However, the above works do not provide a suitable flicker detection and correction scheme for imaging sensor that employ a rolling shutter configuration.